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1. Introduction and background 

A recurring question in the theory of repeated games is to define properly a notion of good .strategy for a player faeing 
an unknown environment. Consequently, in this paper, we are not concerned with the formalisation of strategic 
interactions between rational players, but rather between a decision maker and nature. Not much is known about 
the latter, no assumption is made on its payoff function, its thinking process or its rationality. We take the point 
of view of the former, whose objective is to maximize his/her average payoff in the long run. A naive approach in 
this direction is to assume that the game is zero-sum and to look for optimal strategies. However, the fact that 
his/her opponent might not try to maximize his/her payoff could lead to bad outcomes. A possible definition of 
good strategy for the decision maker has been proposed by Hannan (see Hannan (1957)). It is closely related to 
the concept of regret. After n stages, the regret of the decision maker is the difference between the payoff that he 
could have obtained if he knew in advance the empirical moves of nature and the average payoff he actually got. 
A good strategy for the decision maker may then be defined as a strategy which ensures that, regardless of the 
behaviour of nature, the regret asymptotically goes to zero. Such a strategy is called consistent. Consistent strategies 
are known to exist for a long time and can be constructed, for instance, using so-called block- annealing procedures 
(see e.g. Blaekwell (1954), Foster and Vohra (1993), Foster and Vohra (1998) and Hart and Mas-Colell (2001)). For 
a complete bibliography on the topic, see the last quoted paper. Also, for a recent comprehensive overview about 
consistency in games, see Perchet (2010) (in french). However fictitious play strategies are known to be non-consistent 
(see Fudenberg and Levine (1998)) while smooth fictitious play strategies have been shown to be "almost" consistent 
by Fudenberg and Levine Fudenberg and Levine (1995) (see section 1.2 for a rigorous exposition). In this paper, we 
consider a time- varying smooth fictitious play with a smoothing parameter decreasing to zero, that we call vanishingly 
smooth fictitious play (VSFP). VSFP strategies initially behave like smooth fictitious play and asymptotically like 
fictitious play. The main objective of this work is to answer the following question raised to us by Drew Fudenberg 
and Satoru Takahashi: "are VSFP strategies consistent?" 
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1.1. Notation 

We consider a two-player finite game in normal form. / and L are the (finite) set of moves of respectively player 1 
(the decision maker) and player 2 (the nature). The map tt : / x L — !■ M denotes the payoff function of player 1. The 
sets of mixed strategies available to players are denoted X ~ A(/) and Y = A{L), where 



A(/) := jxelR^ I 1^2;, = l| 



and analogously for A(L). As usual tt is extended to X x y by multilinearity: 

iei leL 

In the following, (ii, i„, ...) (respectively (h, /„, ...)) will denote the sequence of actions picked by player 1 (resp. 
his/her opponent). Let (SI, J^, P) be a probability space, endowed with a filtration {J^n)n- Formally, a strategy for 
player 1 is a choice of an adapted process {in)n on (f2, (J-'„)„,P), Throughout the paper, we assume that the agents 
play independently: specifically, for {i, I) £ I x L, wc have 

P (in+l = ln+1 =^l\Tn)^r (t„+i = i | J"„) P {ln+1 ^ I \ J'n) ■ 

Finally, we call 

1 " 

k=l 

the average moves of player 1 at time n, y„ the average moves of player 2 and 

1 " 

T) ^ ^ 

the average payoff to player 1. 



n 

k=l 



1.2. Consistency, definition and comments 

We now introduce 11 : y M, defined by 

n(?;) max7r(i,y). 

A strategy of the decision maker is consistent if, against any strategy of nature, the average payoff obtained by player 
1 is at least as much as if the sequence of empirical moves of nature was known in advance, and decision maker had 
played a best response against it. More precisely, let us define the average regret evaluation along a sequence of moves 

— (^1 5 ^1 ; • ■ -^71 ; ^n) • 

/ 1 " \ 1 " 

e„ maxTT N, - Un 7r(i™, /„) = n(y„) - 7f„. 

ie/ \ n ^-^ ] n ^-^ 

\ m—l / m— 1 
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Definition 1.1 A strategy for player 1 is said to be consistent if, for any strategy of nature, 

linisupe„ < 0, P — almost surely. 

n 

It is rj-consistent if 

linisupe„ < ry, P — almost surely. 

n 

Given y € Y, we call br{y) the set of best responses of player 1 to y, namely, 

br{y) = Argmax^g^7r(2:,2/). 

The discrete-time fictitious play (FP) process has been introduced in Brown (1951). We say that player 1 uses a FP 
strategy, with prior y^ if, for n > 1, 

P(i„+i = • I Tn) e for(7„), 

where 7„ = ■;;;^yo + ;7^^n- It is well known that this strategy is not consistent. A simple example is given by the 
following (see e.g. Fudenberg and Levine (1998)). 

Example 1.2 Assume that the game is matching pennies, i.e. the payoff matrix of player 1 is given by 

H T 
H f 1 
T yo 1 

and the prior is y^ = (1/3, 2/3). If player two acts accordingly to the deterministic rule heads (H) on odd stages and 
tails (T) on even stages, then player 1 and 2 always play the opposite and the average regret satisfies lim„^oo Sn = 1/2. 

However, jy-consistency can be achieved by small modifications of fictitious play, which are usually called stochastic 
fictitious play strategies. Originally, stochastic fictitious play was introduced by Fudenberg and Kreps in Fudenberg and Kreps 
(1993) and the concept behind this is that players use fictitious play in a game where payoff functions are perturbed 
by some random variables in the spirit of Harsanyi Harsanyi (1973). On the subject, see also Fudenberg and Levine 
(1995), Fudenberg and Levine (1998) or Benaim and Hirsch (1999). In this paper, we adopt another point of view 
and assume that player 1 chooses to randomize his/her moves by adding a small perturbation function to his/her 
initial payoff map ir. 

The class of perturbation functions usually considered (in Fudenberg and Levine (1998) or Hofbauer and Sandholm 
(2002) for instance) is the following: Consider the maps p : Int(X) R such that: 

{Al) the second derivative of p in x, D^p{x) is positive definite on the tangent space of X, 



TX := 



|/i eR^ ■.Y^hi= o| 



(^42) the first derivative of p, defined on Int{X), verifies 

lim ||Vp(a;)|| =+oo. 

x^oX 

We introduce the perturbed payoff function tt defined, ior x ^ X, y E Y and /3 > by 

y, /3) = tt{x, y) + ^P{x). 
Under {Al) and (A2), the function tt enjoys the following property: 
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(i) For all y £ y, /3 > 0, Argvnax^^x^i^j P) reduces to one point and defines a continuous map br from Y x M!^ 
to Int{X). 

The map (y,/3) G F x R!j_ i— >• br(y,/3) is usually called a smooth best response map. 

In our analysis, we also need a little more regularity on the smooth best response map, namely we need the following: 
(m) There exists L > such that the map y ^ br(/3, y) is Lipschitz continuous, with Lipschitz constant 
Hence we replace assumption {Al) by a slightly stronger statement: 
(Al*) There exists A > such that, for any x £ Int{X) and any h G TX , 

{D^p{x)h,h)>\\\hf. 

In particular, notice that (Al*) implies that D^p{x) is invertible and that sup^gj„j(^-, || (L'^p(x))^^ || < j < oo. 
Finally, under assumptions (Al*) and {A2), points [i) and [ii] are checked. 

In the remaining of the paper, we assume that p is a good perturbation function, i.e. a function verifying properties 
{AU) and {A2). 

Remark 1.3 Let p : x £ X i-^ p[x) = — ^ -^j log.T,; he the entropy function. It is a particular case of a good 
perturbation function, and the resulting smooth best response is the so-called logit map, given by 



Efce/exp(/37r(fc,y)) 

Definition 1.4 Player 1 plays accordingly to a smooth fictitious play strategy, with the parameter /3 > (SFP(f3)) 

P(i„+i = t I Tn) = br(y„,/3)„ Vn > 1. 

Tiieorem 1.5 (Fudenberg and Levine, 1995) For any ?; > 0, there exists Po > such that a SFP(f3) strategy is 
rj-consistent for any /? > /3o- 

Smooth fictitious play is closely related to the so-called exponential weight algorithm and also to the follow the 
perturbed leader algorithm (see Cesa-Bianchi and Lugosi (2006), chapters 4.2 and 4.3), even if the link with the latter 
is less obvious. In Hofbauer et al. (2009), the authors discuss the consistency of continuous-time versions of FP and 
SFP. 



1.3. Vanishingly smooth fictitious play 

A related natural strategy is given by the following. Recall that br is a smooth best response function, induced by 
a good perturbation function. 

Definition 1.6 Let {Pn)n ^ sequence going to infinity. The vanishingly smooth fictitious play strategy induced 
by Pn ( and br ) for player 1 is defined by 

P (in+i = I I Tn) = br (l/„, /3„), Vn > 1. 

We use the notation VSFP(/3„) in the sequel. Consistency is not verified for any choice of (/3„))i- If this sequence 
increases too fast, then consistency might fail to hold, as shown by the following example. 
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Example 1.7 Assume that, once again the game is 2-player matching pennies and that nature uses the deterministic 
strategy described in example 1.2. Then, if player one plays accordingly to a VSFP strategy induced by the logit 
map, Pn = n and prior = (1/3, 2/3), we have 

/I 11 1 A J /I 1 1 1 

l2n = - 777; r~rT' n + r~iT ^n'i 72n+l = 77 



^2 6(2n+l)'2 6(2n+l)y ' \2 6(n + l)'2 6(?i + 1) 

After a few lines of calculus (left to the reader) one gets: 



1E('5Z2„ + 1 I ^n) =L(72«,/32n) 



1 



Hence (7r(i2„+i, ^2n+i))n is a sequence of independent random variables taking values in {0, 1}, such that 

limP (7r(z2„+i, /2„+i) = 1) = lim ^ ^ ^2n/3[2n+i) = 1 ^ el/3 = 1/2 - a 
with a > 0. Similarly, (7r(i2n, ^2n))n is a sequence of independent random variables taking values in {0, 1} and 

limP(7r(i2„,^2n) = 1) = , =1/2-5 

with 6 > 0. 

Therefore, consistency is not satisfied for VSFP strategies with /3„ = n since e„ -> (a + b)/2 > 0. 
We now can state our main result 

Theorem 1.8 Any VSFP{Pn) strategy, with /?„ < n'^ for some v <\, is consistent. 

In Benaim et al. (2006), the authors prove the same result as Theorem 1.5 using stochastic approximation methods. 
Specifically, they consider the state variable (a;„, y„, 7f„)„, write it as a stochastic approximation process relative to 
some differential inclusion, and prove that it almost surely converges to the consistency set: 

{(x,2/,7r) : n(y) - tt < 77} . 

This is the approach taken in this paper. In section 2 we show how our state variable can be written as a stochastic 
approximation algorithm, relative to some non-autonomous differential inclusion. A concept of Lyapunov function 
with respect to a set A for non-autonomous systems is introduced in section 3 and, in Proposition 3.7, we establish 
that A attracts the so-called perturbed solutions, under the right conditions. In our specific case, we also prove that 
there exists a Lyapunov function relative to the consistency set. The proof of our main result. Theorem 1.8, is given 
in Section 4. It consists in showing that (x„,y„,7f„)„ is almost surely a perturbed solution with good properties 
and applying the results of Section 3. In the appendix, we provide some general stability results for non- autonomous 
differential inclusions, namely we estimate the deviation of so-called perturbed solutions from the set of solutions 
curves. 
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2. Stochastic approximations 

2.1. A stochastic difference inclusion 

As it was previously mentioned, we are interested in the asymptotic behavior of the state variable w„ := (x„, y„, 7f„) S 
M := X xY X [-|l7r||oo, Ik||oo], where ||7r||oo := maxi,; |7r(z, /)|. We have 

Xn+l -Xn ^ - Ecr((5j„ , 1 I J^n)) = ^ {-^n + br(y„, ^„)) . 

71+1 n + 1 

Writing the analogous recursive formulas for and 7f„, we obtain that 

Vn+l -Vn G ^i^„(-U„), 

n + 1 n + 1 

where 

— the noise sequence 



Un+l = (Vn+l - Vn) - ^{v„+l - Un \ Fn) 



is a bounded martingale difference, 
the set valued map Fn is given by 



K(a;,y,7r) := {(br(y,^„) - x,T - y,7r(br(y,^„),r) - TT, t eY} . (1) 



2.2. Stochastic approximations relative to non- autonomous differential inclusions 

On a more general level, let M and F : R+ x i\if =^ M'' be a set-valued map taking values in the set of non-empty, 
compact, convex subsets of K'^. We say that F is regular if : 

(i?l) s I— 7> F{t,w) is measurable, for each w G M; 

{R2) for any t G M"*", the map w i-^ F{t, w) has a closed graph, i.e. 

{iw,w') e M xM : w' e F{t,w)} 

is closed; 

(i?3) The map F is uniformly bounded, i.e., supj sup^,g^(j ,^-) ||it;'|| < Iji^Hoo < +oo. 
Consider a discrete time stochastic process in M, defined by the recursive formula 

Vn+l - Vn- 7n+lC^n+l G 7n+l -P'n (Wn) , (2) 

where F„ : M ^ R'' is a set-valued map, (7,i)n is a positive sequence, decreasing to and a sequence of 

M-valued random variables defined on a probability space (fi, F, P). Set t„ := X]r=i 'T* ^^"^ to(s) := sup{j | Tj < s}. 
We make the following additional assumptions: 

(SAl) For aU c> 0, 

^e-^/-^" < oo, 

n 
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(5^2) {Un)n is uniformly bounded (by Ht^Hcxj) and 

E {Un+l I Tn) = 0, 

(5^3) The map F -.R+xM ^ M, given by 

Fit,w) :=F„(t)H 

is regular. 

Definition 2.1 If the conditions (5^1), (5*^2) and (5^13) are met, we say that is a good stochastic approxi- 
mation algorithm relative to F. 

Call v{-) the continuous time affine interpolated process induced by and 7(-) (resp. U{-)) the piecewise constant 

deterministic processes induced by (7n)n (resp. {Un)n)'- 

v{Ti + s) = V, + s'^^^ — for s e [0,7i+i], 7(tj + s) := 7^+1 for s £ [0,7i+i[, 

7i+i 

and analogously for U. 

Lemma 2.2 For almost every s G v{-) is differentiate and we have 

v{s) - U{s) £ F(s,u™(s)). 

Proof. We have 

, . Vm{s) + 1 - Vmis) I . 

V\S) = Vrn{s) H [S - Tm(s)) 

7m(s) + l 

Hence, if s ^ {t,j, n G N*}, v{-) is differentiable and 

. / N ''-'m(s) + l ~ ''-'m(s) 
v{s) — — —. 

7m(,s) + l 

Consequently 

v{s) ~U{s) 6 F„(s)(Wm(s)) = F{s,V„i(s))- 

m 

In the sequel, we use the notation tJ(s) :— i'm(s)- Notice that TJ is a piecewise constant map on M^.. 
Let us come back to the particular case of section 2.1, where u„ = (a;„, y„, 7f„) and F„ is given by (1). 

Lemma 2.3 («„)„ is a good stochastic approximation algorithm with step size 7„ = l/n, relative to the map F given 
by F{t,w) := Fm{t){w). 

Proof. We only need to prove that is a regular set-valued map. The fact that F has non-empty compact convex 
values is straightforward, as well as measurability. Also, the map F takes values in a compact set. Thus F is 
uniformly bounded. Given s G K+, we now need to check upper semi-continuity of u i— !■ F[s, v), which is equivalent 
to {{v,w), w G F{s,v)} being closed. Let (a;n,yn,7r„) converge to {x,y,Tr). We then have limn hr {y „, l3m{s)) = 
br(?/,/3„i(5)). Hence, 

lim(br(2/„,/3,„(5)),r„,7r(br(2/„,^,„(5)),r„)) = (br(y, /3„(5)), r, tt (br(y, /3,„(5)), r)) G F{s,x,y,TT). 
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In the particular case where F is an autonomous set- valued map (i.e. F does not depend on t e 1^+), stochastic 
approximation algorithms described above have been studied in Benaim et al. (2005) and they proved that there is a 
deep relationship between the asymptotic behavior of (w„) and the solutions of the autonomous differential inclusion 

w G -F(w). 

In particular, they show that, if there exists a global attractor A for the deterministic dynamics, then the limit set 
of {vn)n is contained in A. 

Unfortunately, in our case, the mean deterministic system associated to our random process {vn)n is a non- 
autonomous differential inclusion, as we will see later on. 

3. Lyapunov functions relative to nonautonomous differential inclusions 
3.1. Perturbed solutions and uniform Lyapunov functions 

Let us consider the non- autonomous differential inclusion 

w(.s) e F(s,w(.s)), s e [a, 6] (3) 

A map w : [a, &] — i> M is a solution of (3) if it is absolutely continuous and, for almost every s G [0,T], w(s) G 
i^(s,w(s)). The existence of solutions from any initial condition is guaranteed under various sets of assumptions, in 
particular for regular F (see Section 5.2 for more details) 

Definition 3.1 ^ map v : R+ M is a perturbed solution of the non-autonomous differential inclusion w(s) S 
F{s, w(s)) if there is a locally integrahle function U : M.^ — >■ such that 

(PSl) V is absolutely continuous, 
{PS2) we have 

rt+h _ 

A{t,t + T):= sup / C/(s)ds -^t^+oo 0, 
/ie[o,T] Jt 

{PS3) v{s) — U{s) G F{s,v{s)) for some measurable map v : M-|- — ^ M such that 

\\v{s)--v{s)\\<5{s), 

with 6{s) Is 0. 

Remark 3.2 Notice that, in the autonomous case, this is Definition {II) in Benaim et al. (2005) 

Proposition 3.3 Let v{-) be the continuous time affine interpolated process associated to a good stochastic approxi- 
mation. Then v is almost surely a perturbed solution, with v(s) — w,„(s) and 6{s) ~ c^(s) (where c is some positive 
constant). 

Proof. This is a direct consequence of Lemma 2.2 and Proposition 4.4 in Benaim (1999). We will provide more 
details in the particular case we are interested in, in Section 4. ■ 

We now define a concept of Lyapunov function adapted to non- autonomous differential inclusions. 

Definition 3.4 Let A be a compact set in M and V be an open neighbourhood of A. A smooth map $ : M+ xV ^ R+ 
is called a uniform Lyapunov function for the non- autonomous differential inclusion (3) with respect to A if the 
following hold: 
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a) we have 

{w eV : 3sn t +00, lim $(s„, ly) 

I n— ^+00 

h) There exists two maps A : M.*^ ^'lO, 1[ and e : M_|_ x M_|_ — > R_|_ wit/i the property that 

lim Am=0 and lim £(t,r) = 0, VT > 0; 

T— >-+oo t— )- + oo 

and, /or ony t > 0, T > and any solution w on [t, t + T], we have 

$(i + s, w(t + s)) < A(s)$(t, w(t)) + e(i, T), Vs G [0, T]. 
If \^ = M then $ is called a global uniform Lyapunov function. 

Remark 3.5 Assumption a) is checked in particular if the somewhat more explicit condition is verified: 
a') there exists a continuous map g : V ^ R+ such that 

A = {weAI : g{w) = 0}, \\g{w) - $(s,w)|! ->,,^+oo 0, 

uniformly in w £ V . 

The following lemma will be useful to prove the main result of this section, namely Proposition 3.7. 
Lemma 3.6 Let {^k)k>koj {^k)k>ko {rik)k>ko be positive sequences of real numbers such that < Afe < 1 and 
(i) for any k > kg 

$fe+i < Afc<I>fe +77fe+i; 

(a) for k > ko + 1, denoting Hk := ^'iZlo^i o.'^d Hk = Hk Yln=Q H^^rji, we have hmfe^oc. Hk = linifc^oo Hk = 0. 
Then limfc^oo $fc =0. 

Proof. Without loss of generality, we assume that fco = 0. A simple recursive argument yields 

< Hk + 

and the proof is complete. H 

We say that $ is uniformly Lipschitz if there exists L$ > such that, for any s > and w, w' G M, 

|$(s, w) - $(s, w')\ < L^\\w - w'\\. 

We now need to define Lipschitz continuity for non-autonomous set- valued maps: call dn the Hausdorff distance, 
given by 

dniE, F) = max < sup d{x, F), sup d{y, E) 

Recall that dn is a pseudo-metric on the set of non-empty subsets of M and a metric if we restrict to the non-empty 
compact sets of M. We say that F is Hausdorff continuous if it is continuous with respect of the Hausdorff metric: 

lim dH{F{t,w),F{t^,wo))=0. 
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If F is Hausdorff continuous, we call it L-Lipschitz. for an intcgrablc function L : [a, b] M_|- if 

dH{F{t,w),F{t,w')) < L{t)\\w - w'll, for a.e. t e [a, 6], W w,w' 

We now state the main result of this section. Corollary 5.7 plays an important role here, as it gives upper bound 
for the deviation of perturbed solutions from actual solutions of the deterministic system. For convenience of the 
reader, we chose to postpone this technical result to Section 5.2. 

Proposition 3.7 Assume that v is a perturbed solution relative to a regular Lipschitz map F (with L : — > j 
and that ^ is a global uniform Lyapunov function with respect to a compact set A and the differential inclusion (3). 
Assume also that there exists a sequence of positive real numbers (Tk)k such that 

(i) Sk ■■= Y.i=i Ti +00, 

{ii) there exists and a sequence {rk)k such that, for any k > kg 

R{Sk,Sk+i) < rk, 

with R defined by ( 6) in Corollary 5. 7, 

(iii) $ is uniformly Lipschitz, with constant L,^, 

(iv) denoting Hk := n^L^. , ^A(Ti) and r]k := £{Sk-i,Tk) + L$rfe_i, we have 



lim iffc Vi/ri,,,, =0. 



fc— > + oo 

?'— 1 

Then the limit set of v, jC{{v{s))s>o) {v* : 3sn f +cxd, limn^(sn) — v^} is contained in A. 

Proof. First, by Corollary 5.7, for any k G N, there exists a solution on [Sk,Sk-\-i] such that w^(5'fc) — v{Sk) and 

sup ||2;(s)-w^(5)|| <i?(5,.,5,+i). 
se[Sfc,Sfc+i] 

By (ii) the sequence of solutions curves {'w^)k>ko is such that 

sup \\v{s) - w''(s)|| < rk- 

se[Sk,Sk+i] 

On the other hand, by definition of $ and w'', we have 

w''(5fc+i)) < A(Tfc+i)$(5fc, w^C^fe)) + e{Sk, Tk+i). 
Hence, by (Hi) and (iv), for any k > ko, 

<^{Sk+i,v{Sk+i)) < <i>{Sk+i,w''{Sk+i))+L^\\v{Sk+i)-w''{Sk+i)\\ 

< X{Tk+iMSk,v{Sk)) + L.j,rk + e{Sk,Tk+i) 

= X{Tk+iMSk,v{Sk))+Vk+i 

Clearly, Hk 0, by definition on A. Calling $fc ^{Sk,v{Sk)) and Xk ■= X{Tk+i) we have — by Lemma 3.6. 
Now let be a limit point of v{s): = lim„ w(s„), for some sequence s„ tn +oo. Call k{n) := sup{fc G N : Sk < Sn}- 
For n large enough, k{n) > ko and 

$(s„,u(s„)) < A(s„ - S'fc(„))$(S'fc(„),u(S'fe(„)) +L$rfe(„) +e(S'fe(„),s„ - Sk(n)) -^n^+oo 0. 

We therefore have 

$(Sri,W*) < ^{Sn,v{Sn)) + L^\\v^ - ^nW -^n^+oo 0. 

Consequently £ A and the proof is complete. I 
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3. 2. A Lyapunov function for the differential inclusion induced by (1) 

We now focus on the particular case of Section 2.1 and prove that there exists a global Lyapunov function with 
respect to the so-called consistency set. 

Theorem 3.8 Let A = {(;r,y, tt) G AI \ Hijj) — tt < 0}. There exists a global uniform Lyapunov function $ relative 
to the compact set A and the non- autonomous differential inclusion 

vf{s) e F{s,w{s)). (4) 

Proof. Wc prove that properties a') and b) (of respectively Remark 3.5 and Definition 3.4) hold. 
Let $ : E+ X 7\/ ^ IR+ be defined by 

a>(. r „ tt)- i n(y,/3™(s)) -TT if n(y, ) > TT 
ns.x^y,7r)-<^ ^ ifn(y,/3„(,))<^. 



where 



n : y X K; ^ M, (y, /?) ^ max ^(x, y,P)=n (br(y, /?),?/, /3) . 



Notice that 

A = {{x,y,TT) : g{x,y,TT) =0} and \\g{x,y,TT) - ^{s,x,y,TT)\\ ^s~^+oo 

uniformly, where g(x, y, tt) := max{0, n(y) — tt}. Let t and T be positive real numbers and w(s) := (a;(s), j/(s), 7r(s)) 
be a solution of the non-autonomous differential inclusion (4) on [t,t + T], such that 7r(s) < n(y(s), ^^(s)). Thus 

y(s) = t(.s) - y(s), Tr(s) = 7r(br(y(s), t(s)) - 7r(s), 

where t(s) G F, Vs. Let 

*(s) := $(s, a;(s), 2/(s), 7r(s)) = ir {hr{y{s), P,n{s)), y{s), I3„i(,s)) - 7r(s). 

Recall that Pm{s) is piecewise constant on [t,t + T]. Hence, for almost every s e [t,t + T], we have 

4'(s) = TT (br(y(s),/3„(^)),y(s),/3,„(5)) - 7t(s) 

= TT (br(y(s),/3,„(5)),r(s),/3,„(s)) - tt (br(y(.s), /3,„(5)), y(s), /3,„(5)) 
-7r(br(y(.s),/3„,(^)),T(s)) -|-7r(s) 

< + -^p(br(y(s), < + 

Pm(s) Pm(s) 

where we recall that p denotes the perturbation function. The first equality is obtained using the cnveloppe theorem 
and the fact that tt is linear in its second argument. Thus, by an application of Gronwall's lemma, we obtain 

+ T) < e^^*(t) + ^ 



Consequently, $ is a global uniform Lyapunov function with respect to A, which proves the result. 
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4. Proof of Theorem 1.8 

We are now ready to prove our main result. We already proved that the interpolated random process induced by 
{vn)n is almost surely a perturbed solution of the differential inclusion (4) with 5{s) = 07(5), and that there exists a 
global uniform Lyapunov function with respect to 



see respectively Proposition 3.3 and Theorem 3.8. Therefore we now check that the assumptions of Proposition 3.7 
hold. Be aware that we have not used the particular form of the parameter sequence (/3„)„ so far. Recall that /3„ = n", 
for some v G (0, 1). 

Notice that 7„ = 1/n. Therefore we have r„ ^ logn and m{s) — ©(e")^. Recall that, given positive real numbers 
t and r, A(t, t + T) denotes the random variable 



Although the quantity ¥{A{t,t + T) > a) always vanishes under assumptions (SAl) and (SA2), we need to know a 
little more. The next lemma (proved in Duflo (1997) or Benai'm (1999) for instance) gives an upper bound of this 
quantity. 

Lemma 4.1 There exists positive constants C and C (depending on \\U\\oo) such that, for any a > 0, 



The set-valued map F is regular and i(-)-Lipschitz, with the same Lipschitz constant as the map (s, y) 1— > 
hr {y , f3m{s))- Hence L{s) = iy/3,„(s), for some constant L (see Section ??). Hence, we can assume without loss of 
generality, that L{s) < e'^^ (up to choosing i/' > v). In the next proposition, we see that assumptions (i) and (ii) of 
Proposition 3.7 hold, if we choose Tk = (fk)'^ . 

Proposition 4.2 If we choose Tk := (vk)^^ there exist some constant r > 1 with the property that, with probability 
one, there exists fco G N such that points (i) and (ii) of Proposition 3. 7 are verified for v, with rk = k^^ 

Proof. Point (i) clearly holds. We now need to prove [ii). In this particular case, the quantity R{Sk, Sk+i) satisfies 



A = {{x,y,TT)eAI\n{y) 



7r<0}, 




■ t+h 



sup / 

he[0,T] Jt 





By our choice of the sequence {Tk)k, exp(z^S'fc) < exp (1 + log k) < 3k. Hence 




for some constant Co which depends on ly. Additionally, j{Sk) < ^er^" < 2^-1/". Hence 



C7(5fc)exp(rfc+ie''^''+i) < 



more precisely, ^-j^e" < m{s) < e" — 1 
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Choose r e (1, ^^T")- -^^ Lemma 4.1, 

A(5fe,5fe+i)cxp^y^ L(^)d^J>_J < cexp' 

< C exp 

< Cexp 



C'Cov-^{k + l)-^ 

_^-2r+l + l/iy 



C[ 

for some positive constant Now, since r < l/v, we have for fc large enough 

c7(5fe)exp(Tfc+ie''^''+0 < 

Consequently, if we call Ak the event 



|(A(5fe, + C7(^fe)) exp i^j'^J^' L(r)drj > 1 

then 

'(Afc) <Ccxp' 



By an application of the Borcl-Cantelli lemma, with probability one, there exists fco G N such that, for any fc > N, 



R{Sk,Sk+i) < (A(5fe,5fc+i) + c7(^fc))exp ( / Lir)dr \ < 



1 



which yields the result. 



Remark 4.3 By similar arguments, we can also prove the following: Assume that F is L-Lipschitz, with L(s) < Ls. 
Then there exist T > 0, and r > such that, with probability one, there exists fcg G N with the property that points 
(i) and (ii) of Proposition 3.7 are verified for v, with T^ — T and = e^^*^. 

Consequently, points (i) and {ii) of Proposition 3.7 arc almost surely satisfied for k > ko, with Tk = (lyk)"^ and 
fk = k~^ , r > I. We now need to check points (Hi) and (iv). 

Let 6 be a positive constant and consider the map (jj : Y x [— ||7r||oo, ||7r||oo] ^ R+, given by 



(/)(y,7r) = 



fliy,b)-7r ifn(y,&)>7r 
if n(y, b) < TT. 



Let (y, tt) be such that n(j/, b) > n. Then, by Lemma 6.2 in Bcnaim ct al. (2006) (see also Fudenberg and Levine 
(1999)), we have 

d d 

—(j){y,Tr){h) = — 7r(br(y,6),2/,6)(/i) 

= 7r(br(y,6),/i) 
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and 
Thus 

\(f>{y,T^) - '/'(y',7r')l < Iklloolly - y'W + k - tt' 

and (j) is Lipschitz with Lipschitz constant that docs not depend on b, which means that the map v i— > ^{s, v) is 
uniformly Lipschitz. 

We now prove point (iv). By Theorem 3.8, $ is a global uniform Lyapunov function relative to 

A = {ix,y,TT)eAI\U{y)-7r<0}, 
with A(r) = e"^ and e{t, T) = j^. Hence 

rjk+i=L^k + < L$fc +c— — , 

Pm(Sfc) K 

for some positive constant c. We have rji < oo and Hk ~ ^i=fco — 0{k~^^'^). Thus point (iv) is checked (see 
point b) of Lemma 5.1 for a proof). 

As a consequence, Proposition 3.7 applies and 

/:((t;(s)),>o) C A 

almost surely. In particular 

limsupe„ < 0, almost surely 

n 

and Theorem 1.8 is proved. 
5. Appendix 

5.1. Sufficient conditions for Lemma 3.6, (ii) to hold 

Lemma 5.1 Point (ii) of Lemma 3.6 is verified in the following cases: 

a) Afc = A < 1 and hm^^oo Vk = 0, 

b) limfe^oo Hk = and J^i Vi < 

Proof. For point a), Hk = A'^ and we have 



(fe k+k' 
i=l i=k+l 
k'-l 

< a'' max 7], + rik+i- 



i=0 
1 



i=0....,k 1 - A 
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which gives the result. 

For the second point, remember that {Hk)k is a decreasing sequence. Hence 

(A: k+k' \ 

1=1 i=k+l J 

Ck \ +00 

1=1 / i=A:+l 

Given e > 0, by choosing k large enough, the second term is smaller than e. Then we can pick k' large enough so 
that the first term is also smaller than e and the proof is complete. ■ 

5.2. Stability of one-sided Lipschitz differential inclusions 

Let M C M.'^. Consider a set-valued map F : R+ x M =^ M taking values in the set of non-empty, compact, convex 
subsets of M. Given a < b, let us consider the non-autonomous differential inclusion (3): 

W(s) e F(s,w(s)), s e [a,b]. (5) 

For A C M we let F-'^{A) = {(s, w) e [a,b] x M : F{s,w) n A 7^ 0}. Wc say that F is measurable if F~^{A) is 
measurable, for any closed set A C M. It is upper semi- continuous (USC) (rcsp. lower semi- continuous (LSC)) if, 
for any closed (resp. open) set A C M, F~^{A) is closed (rcsp. open) in [a,b] x M. If M is compact, F is upper 
semi-continuous if and only if its graph is closed. 
Wc now introduce a regularity condition: 

Definition 5.2 (Relaxed One-sided Lipschitz) we say that the set-valued map F is Relaxed One-sided Lipschitz 
(ROSL) on [a, b] x M if there exists an integrahle map L : [a, b] — > M such that, for any t, t' in [a, b] w, w' G M and 
any y G F(t,w) there exists y' £ F{t',w') with 

<w' ~w\y' -y>< L{t)\\x' - xf , Vt G [a,b]. 

Remark 5.3 If F is L{-)-Lipschitz then it is L{-)-ROSL. 

The question of existence of solutions to (3) has been studied extensively. One of the first result on the topic was 
proved by Filippov (see Filippov (1971)) and says that if F(-, •) is Hausdorff continuous on any closed set of [a, 6] x M 
then, for any wq G M, there exists a solution w(-) of (3), with w(a) = xq. Under less restrictive assumptions, the 
same result still holds (see Olecli (1975); on the topic, see also Himmelberg and Van Vleck (1986)). 

Theorem 5.4 (Olech, 1975) Assume that F is regular. Then there exists a solution w(-) of (3), with w(a) = wq. 

The following result will prove useful to establish Theorem 5.6. 

Lemma 5.5 Let y be a continuously differentiable function on [a, b] and f , g be non-negative, continuous maps. If, 
for every s G [a,b], \\y{s)\\ < f{s)\\y{s)\\ + g{s) then 

\\y{s)\\ < ||y(a)||exp ^ /(T)dT^ +^ .g(^i)exp ^ /(T)dT^ ds 
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Proof. Notice that 

||y(s)|| < ||2/(a)||+ f \\y{u)\\du<\\v{a)\\+ f g{u)du+ f f{u)\\y{u)\\du 

and apply the integral form of Gronwall's lemma. ■ 

In the remaining of this section, we assume that F is regular. The set of solution trajectories on [a, 6] (resp. starting 
in Wo) will be labelled 5(a, 6) (resp. iS(zi;o, a, 6)). 

Theorem 5.6 Let W : [a, b] M be an absolutely continuous function such that there exists a measurable map 
V : [a, b] — > M and a bounded measurable map r : [a, b] M-|_ which satisfy, for almost every s G [a, b], 

d{W{s),v{s)) < r(s), Wis) G F{s,v{s)). 

Then 

a) if F is ROSL with respect to the integrable function L, then there exists a solution w : [a, 6] M of (3) such 
that w(a) — W{a) and 



sup II w(s) -VF(s)f < [ a{s)expU f i(T)dT ) ds, 



where a{s) = 4:L{s)r^{s) + 4r(s) l|F||oo- 
b) if we now assume that F is Lipschitz continuous, with respect to L then the conclusions of a) trivially still hold 
and w can also be chosen such that 

sup ||w(.s) - VK(s)|| < / r{s)L{s)cxp( [ L(T)dT ) ds. 

Proof. We prove the first point. Consider the set-valued map G : [a, b] x M ^ M given by 

G(s, x) := |u e F(s, w): {w ~ W{s) \ v - W{s)) < 2L{s)\\w ~ W^(s)f + ^a{s) 

For any {s,w), the set G{s,w) is non-empty. Indeed, by the ROSL condition, since W{s) G F{s,v{s)), there exists 
V G F{s,w) such that 

{w - v{s) I V - W{s)) < L{s)\\w - vis)f. 

Hence we have 

{w-W{s) \v^W{s)) < Lis)\\w~v{s)f + \\v{s)^Wis)\\{\\v\\ + \\Wis)\\) 

< 2L{s)\\W{s) - + 2L{s)r{sf + 2r(s)||F||oo 

= 2L{s)\\W{s)-wr + ^a{s). 

Now clearly, the set G{s,w) is compact and convex. The map w f-^ G{s,w) has a closed graph, for any s G [a,b]. 
Finally It is measurable in s since every map involved is measurable. Consequently, there exists a solution to the 
non-autonomous differential inclusion 

w(s) G G(s,w(s)), 
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with initial condition w(a) = W{a). In particular, w is a solution of (3) and wc also have, for almost every s 

{w{s) - W{s) I w(s) - W{s)) < 2L{s)\\Wis) - vir(s)f + ^a(s). 

Hence , for almost every s, we have 

^ ||w(s) -M^(s)f = 2{w{s)~W{s)\vf{s)-W{s)) 
as 

< AL{s)\\W{s)~w{s)\\^ +a{s) 

and point a) follows from the differential form of Gronwall's lemma. 

When the Lipschitz continuity holds, let us consider the set- valued map H : [a,b] x A/ =^ M given by 



H 



{s,w) := |w G F{s,w) : \\v ~ W{s)\\ < L{s)\\w - W{s)\\ + L{s)r{s)^ 



The fact that H has non-empty values follows from Lipschitz continuity: given s and w, since W{s) € i^(s,w(s)), 
there exists v G F{s, w)) such that 

\\v - Wis)\\ < L{s)\\w -vis)\\ < L{s) {\\w ~ W{s)\\ + \\W{s) ^v{s)\\) . 

Hence v € H{s,w) ^ 0. Also H(s,w) is convex and compact, the map w t-^ H{s,w) has a closed graph and 
s I— >■ H{s,w) is measurable. Thus, there exists a solution x to the non-autonomous differential inclusion 

w(s) e i7(s,w(s)), 

with initial condition w(a) = W{a). In particular, w is a solution of (3) and we also have, for almost every s 

||W(s) - W{s)\\ < L(s)|[w(s) - W{s)\\ + L{s)r{s) 
By Gronwall's lemma (see Lemma 5.5), we then have 

sup ||w(s) - VK(s)|| < / L(s)r(s) cxp[ / L(T)dr ) ds 



se[a,b] 

and point b) is proved. I 

Corollary 5.7 Let v : [a, 6] — > M be an absolutely continuous map. Assume that there exist measurable maps 
V : [a, b] — > M , S : [a, b] — >■ R+ bounded and U : [a, b] — M integrable such that, for almost every s G [a, b], 

v{s)~U{s) e F{s,v{s)), ||w(s) -TJ(s)|| < d{s). 

Then if F is L{-)- Lipschitz, there exists a solution w on [a,b] such that w(a) = v{a) and 

sup \\v{s) ~ w{s)\\ < R{a,b), 

se[a,b] 

where 

i?(a, b) = A(a, b) exp ( / L(r)(ir ) + sup S{s) | cxp ( / L{T)dT ) - 1 ) (6) 

\Ja J se[a,b] \ \Ja J J 

and A{a,b) ^ snp,^^^^^ \\ £U{T)dT\\. 
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Proof. Define W : [a, b] -> M by 

W{s) := v{s) - I U{T)dT. 

Clearly, W is absolutely continuous and, for any s for which v is differentiable, we have W{s) ~ v{s) — U{s) S 
F(s,TJ(s)). Additionally, 



\\W{s)-v{s)\\<\\v{s)-v{s)\\ + \\ / U{r)dr\\<5{s) + 



U{T)dT 

By a direct application of Theorem 5.6 with r(s) = 5[s) + || J^* [/(T)c?r|| , there exists a solution w such that w(a) 
W{a) = v{a) and 

sup ||w(s)-w(s)|| < A{a,b)+ [ L{s)(s{s) + \\ [ U{T)dT\\) cxp I [ L{T)dT] ds 

< A{a,b) + { sup S{s) + A{a,b)) f L{s)exp(f L{T)dT\ ds < R{a,b). 

siE[a,b] Ja \J s J 
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